Skip to main content

Avro actions

1. About Avro Client​

This action initializes the connection configuration.

The following properties are required:

  • Brokers: A comma separate list, for example:
    • mybroker1.test.io:9092,mybroker2.test.io:9092,mybroker3.test.io:9092
  • avroHost: schema registry host, for example:
  • Client ID
  • Log Level:
    • Info (default)
    • Warning
    • Debug
    • Error
    • None
  • Connection Timeout
    • default is 3000

2. About Avro Consumer​

This is an Avro-configured consumer that exclusively accepts messages that conform to the schema registry's configuration. Its proper functioning necessitates the presence of all the client configuration connectors with appropriately configured client connections.

3. About Avro Deserialize​

This connector necessitates the specification of a schema registry host and employs it to decode (i.e., deserialize) encoded (i.e., serialized) messages.

4. About Avro Encode Action​

The Avro Encode connector action is a data transformation process that converts data from other formats into the Avro format. Avro is a data serialization system that provides a compact, fast, and efficient way to exchange data between different systems. The Avro Encode connector action can be used in ETL (Extract, Transform, Load) workflows to prepare data for storage or processing in a Hadoop ecosystem. This action is useful when you need to convert data from other formats such as CSV or JSON into Avro format. Once the data is in Avro format, it can be easily integrated with other tools in the Hadoop ecosystem, such as Spark or Hive.

Application of Avro Encode Action

The Avro Encode connector action has various use cases in the data engineering and big data world. Some of them are:

  1. Data Integration: The Avro Encode connector action can be used to convert data from different sources into the Avro format, which is easily integrable with various tools in the Hadoop ecosystem. This allows for seamless data integration, making it easier to process and analyze data.
  1. Data Storage: Avro is designed to be a compact and efficient data serialization system. Storing data in the Avro format can help in reducing storage requirements, as well as make it easier to query and retrieve data.
  1. Data Processing: The Avro format is optimized for data processing, making it ideal for use in Hadoop ecosystems. Once data is in the Avro format, it can be processed by various tools like Spark, Hive, Pig, etc., for analytics and machine learning.
  1. Stream Processing: The Avro format is also used in stream processing systems like Apache Kafka, where it is used to serialize and deserialize data between producers and consumers. The Avro Encode connector action can be used to convert data from other formats into Avro format before sending it to Kafka.

In summary, the Avro Encode connector action is a useful tool for converting data from different formats into the Avro format. It has a wide range of use cases in data integration, storage, processing, and stream processing systems.